Particle Gibbs with Ancestor Resampling for Probabilistic Programs

In this paper we develop an implementation of a recently proposed inference algorithm known as particle Gibbs with ancestral resampling (PGAS) [Lindsten et al., 2012] for higher-order probabilistic programming languages. Higher-order probabilistic languages such as Church [Goodman et al., 2008], Venture [Mansinghka et al., 2014] and Anglican [Wood et al., 2014] allow statistical models to be represented as programs. Evaluation of a program F , which we will here represent as a sequence of statements F = f1:N , instantiates random values x for some subset of the expressions, which can be thought of as a sample from a prior p(x |F ). A program F [x] where all sampled values are substituted as constants is once again deterministic. This substitution also forms the basis for posterior inference, which samples the unconstrained variables from p(x |F [y]), where constants y are substituted for some random values in F [y]. In higher-order languages it is straightforward to define non-parametric models, which can instantiate a arbitrary numbers of variables, or specify model structures recursively in terms of a generative grammar. This offers a greater flexibility relative to declarative systems such as Infer.NET [Minka et al., 2010] and STAN [STAN Development Team, 2014], which restrict the model definition syntax in order to omit higher-order functions and recursion. At the same time this expressivity makes it difficult to characterize the support XF = {x : p(x |F ) > 0}, since neither the number of random values or their conditional dependencies are necessarily fixed for all possible execution histories.